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patent application entitled, "Printer With Audio or Video Receiver, Recorder, and Real- 
Time Content-Based Processing Logic," filed March 30, 2004, Attorney Docket No. 
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BACKGROUND 

Field of the Invention 

[0003] . This invention relates generally to printers that have embedded functionality 
for printing time-based media, and in particular to printing time-based media that results 
in a combination of a printed output and a related electronic data output. 
Background of the Invention 

[0004] A conventional printer can receive documents or other data in a number of 
formats and then prints the contents of those documents or data in accordance with the 
proper format. But while conventional printers can print documents in a wide variety of 
formats, these printers are fundamentally limited in their ability to reproduce different 
kinds of media. For example, it is standard technology for a printer to produce images of 
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static text, pictures, or a combination of the two. But because these printers print onto 
paper or another similar fixed medium, they cannot record the nuances of time-based 
media very well. 

[0005] What is needed therefore is a printer that is equipped to print time-based 
media without the limitations of conventional printers. It is further desirable that such a 
print be able to perform at least some of the necessary processing itself rather than require 
an attached computer or other device to perform all of the processing. 
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Summary of the Invention 
[0006] To satisfy the needs unmet by conventional printers, a multifunction printer 
enables the printing of time-based media. In a typical hardware configuration for such a 
multifunction printer, a printer includes a print engine that produces a paper or other 
printed output and one or more electronic devices the produce a related electronic output. 
Together, the printed and electronic outputs provide an improved representation of the 
time-based media over that of a convention paper printer. 

[0007] In one embodiment, a printer includes an interface for receiving time-based 
media and a media processing system coupled to the interface to receive the time-based 
media. Based on any of a number of desired applications, the media processing system 
determines a printed representation of the time-based media and an electronic 
representation of the time-based media. A printed output system in communication with 
the media processing system receives the printed representation and produces a 
corresponding printed output. Similarly, an electronic output system in communication 
with the media processing system receives the electronic representation and produces a 
corresponding electronic output. In this way, the printer creates a representation of time- 
based media by producing a printed output and an electronic output. 
[0008] In various embodiments, the printer includes different combinations of 
embedded devices that allow for various mechanisms for receiving the media and various 
mechanisms for creating the electronic and printed outputs. For example, the interface 
may include a single communication interface, a network interface, a removable media 
storage device reader, a video input device (such as a DVD reader or a video cassette 
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reader), an audio input device (such as a CD reader or an audio cassette reader), a media 
broadcast receiver, a radio receiver, screen capture hardware, a video and/or audio 
recorder, or any of a number of different types of devices that can receive time-based 
media. Similarly, the electronic output system may write the electronic representation to 
one or more different types of removable media storage devices, such as a DVD, a video 
cassette tape, a CD, an audio cassette tape, a flash card, a computer disk, an SD disk, or 
another computer-readable medium. The electronic output system may also include a 
disposable media writer, a self-destructing media writer, a video display, an audio 
speaker, a driver for a speaker system (such as an embedded MIDI player), or an 
embedded web page display. In this way, a multifunction printer can be configured to 
have any of a large number of combinations of embedded subsystems, allowing various 
embodiments of the printer to meet the needs of many different applications. 
[0009] Because of the great many combinations of input and output devices possible 
for the multifunction printer, the printer may include embedded hardware, software, or a 
combination thereof for performing a wide variety of different operations on the media. 
In this way, the printer can be configured to produce various types of printed and 
electronic outputs based on received media to meet the needs of different applications. 
To solve various problems, in embodiments of the printer, the media processing system 
includes one or more of an embedded multimedia server, an embedded audio encryption 
module, an embedded video encryption module, an embedded audio sound localization 
module, and an embedded video motion detection module. 
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[0010] These different tasks may be performed on the printer by the media processing 
system, or partially on the printer by the media processing system in conjunction with one 
or more electronic devices capable of performing some of the required processing steps. 
The printer can thus balance the required processing of the media between the printer and 
one or more connected electronic devices, such as a personal computer or an external 
network service. By conducting at least some of the processing on the printer, the printer 
relieves at least some of the processing load on external devices that the printer's 
additional functionality may require. 
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Brief Description of the Drawings 
[0011] FIG. 1 is a schematic diagram of a printer in accordance with an embodiment 
of the invention. 

[0012] FIG. 2 is a flow diagram of the operation of the printer in accordance with an 
embodiment of the invention. 

[0013] FIG. 3 is a schematic diagram of various media source input subsystems of the 
printer, in accordance with embodiments of the invention. 

[0014] FIG. 4 is a schematic diagram of the printer's printed output system, in 
accordance with an embodiment of the invention. 

[0015] FIG. 5 is a schematic diagram of various electronic media output systems of 
the printer, in accordance with embodiments of the invention. 
[0016] FIG. 6 is a schematic diagram of various media processing systems of the 
printer, in accordance with embodiments of the invention. 
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Detailed Description of the Preferred Embodiments 
[0017] Various embodiments of a multifunction printer enable the printing of time- 
based media in a useful and intelligent format. To create a representation of this time- 
based media, the printer produces a printed output and a related electronic output, which 
together provide a representation of the received media. Depending on the desired 
application for the printer, the printer may include any number of devices for receiving 
the media, printing the printed output, and producing the electronic output. Therefore, a 
number of embodiments of the printer are described herein to show how such a printer 
can be configured in a virtually limitless number of combinations to solve or address a 
great number of needs that exist. 
Printer Architecture 

[0018] FIG. 1 is a high-level diagram of one embodiment of the printer 100. Printer 
100 includes a media source interface 105, a user interface 1 10, a printed output system 
1 15, an electronic output system 120, and a media processing system 125. Capable of 
receiving time-based media 150, the media source interface 105 can take a variety of 
forms and may include one or more devices that can receive media data or create media 
data by observing a media event. Similarly, the printed output system 1 15 and the 
electronic output system 120 can take a variety of forms and may each include one or 
more devices that can produce, respectively, a printed output 160 and an electronic output 
170. The user interface 110 may include a display system, software for communicating 
with an attached display, or any number of embodiments described in co-pending U.S. 
patent application entitled, "Printer User Interface," to Hart et al., filed March 30, 2004, 
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Attorney Docket No. 20412-08455, which application is incorporated by reference in its 
entirety 

[0019] In one embodiment, the media processing system 125 includes a memory 130, 
a processor 135, and one or more embedded functionality modules 140. The embedded 
functionality modules 140, which are described in more detail below, may include 
software, hardware, or a combination thereof for implementing at least a portion of the 
functionality of the multifunction printer 100. The media processing system 125 is 
coupled to the media source interface 105 and the user interface 1 10, allowing it to 
communicate with each of those devices. The media processing system 125 is also 
coupled to the printed output system 1 15 and to the electronic output system 120 for 
providing the appropriate commands and data to those systems. 
[0020] FIG. 2 shows an overview of a generalized process in which the printer 100 
creates a representation of time-based media data 150, in accordance with one 
embodiment of the invention. The printer 100 first receives 205 time-based media 150 
from an external source with the media source interface 105. This media 150 may be 
received as digital or analog data, or it may be an observable event that the interface 105 
records as digital or analog data. Coupled to the interface 105 to receive the media 150, 
the media processing system 125 processes 210 the media data to generate printed and 
electronic outputs. This processing 210 is performed in accordance with the intended 
functionality of the printer 100, and examples of different operations are described in 
greater detail below. 
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[0021] In one embodiment, the printer 100 includes a user interface 1 10 to allow a 
user to preview the generated outputs. If 215 the user desires to refine the processing, the 
user can enter commands, which the printer 100 receives 220 by way of the user interface 
1 10. Based on the user's commands, the printer 100 then repeats the processing 210 of 
the media data to generate new printed and electronic outputs. This refinement process 
can be iterated until the user is satisfied with the printed and electronic outputs. When 
this occurs, the user will indicate that the printing should commence, for example, by 
invoking a print command with the user interface 1 10. The media processing system 125 
then sends the generated printed output to the printed output system 1 15 and the 
electronic output to the electronic output system 120. The printed output system 1 15 then 
creates 225 a printed output 160, and the electronic output system 120 creates 230 an 
electronic output 170. 

[0022] It can be appreciated that this generalized description of a multifunction 
printer lends itself to a great number of specific configurations and applications. 
Accordingly, examples of the possible configurations, applications, and particular 
components are further described. 

Media Source Interface 
[0023] The media source interface 105 can be designed to accommodate any suitable 
type of time-based media. Because of the great variety of types and formats of media 
data, the media source interface 105 may take any number of forms to accept any time- 
based media that a user might wish to print. FIG. 3 illustrates some examples of different 
interfaces 105 by which the printer 100 can receive media data from an external source. 



- 10- 



20412/08340/SF/51 16435. 1 



In particular implementations, the printer 100 may have only one or only a subset of these 
types of interfaces 105, and in addition may have other types of interfaces not shown. 
[0024] As shown in FIG. 3, the printer 100 may include a single communication 
interface 305 that allows the printer 100 to be communicatively coupled to another 
electronic device. Depending on the desired input, the interface 305 may allow the 
computer to communicate with a wide variety of different electronic devices that can 
provide the printer 100 with time-based media to print. Without intending to limit the 
types of devices, the interface 305 may allow the printer 100 to received media data from 
external sources such as computer systems, computer networks, digital cameras, video 
cameras, media Tenderers (such as DVD and CD players), media receivers (such as 
televisions, satellite receivers, set-top boxes, radios, and the like), external storage 
devices, video game systems, or any combination thereof. The connection type for the 
interface 305 can take a variety of forms based on the type of device that is intended to be 
connected to the printer 100 and the available standard connections for that type of 
device. For example, the interface 305 may comprise a port for connecting the device 
using a connection type such as USB, serial, Fire Wire, SCSI, IDE, RJ1 1, optical, 
composite video, component video, or S-video, or any other suitable connection type. 
[0025] In another embodiment, the printer 100 includes a wireless interface 310. As 
illustrated, the wireless interface 310 allows the printer 100 to receive media data from a 
wireless device external to the printer 100. The wireless interface 310 may allow the 
printer 100 to communicate with any number of wireless communication systems, such as 
wireless components on a home or business network, cellular phones and other portable 
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wireless devices, satellites, satellite dishes, and devices using radio transmissions. 
Depending on the types of external devices with which the printer 100 is desired to 
communicate, the wireless interface 310 may comprise hardware and/or software that 
implements a wireless communications protocol, such as that described in IEEE 802.1 1 
or the Bluetooth standard. 

[0026] In another embodiment, the printer 100 receives media data from a removable 
media storage reader 315 that is built into the printer 100. The removable media storage 
reader 315 may be configured to accommodate any type of removable media storage 
device, such as DVDs, CDs, video cassette tapes, audio cassette tapes, floppy disks, ZIP 
disks, flash cards, micro-drives, memory sticks, SD disks, or any other suitable type of 
media storage devices. Moreover, the printer 100 may have a plurality of removable 
media storage readers 315 to accommodate multiple types of media storage devices. 
[0027] In another embodiment, the printer 100 includes an embedded video recorder 
330. In this embodiment, the external source of media data is a series of images captured 
by the embedded video recorder 330. The video recorder 330, such as a camera, CCD, or 
other suitable mechanism for capturing a sequence of images, converts a scene into a 
suitable electrical format, such as that described in the MPEG, H.263, or H.264 standards. 
Optionally, an analog-to-digital converter 335 converts an image directly from the video 
recorder 330 into a digital format, which is then provided to the media processing system 
125. In yet another embodiment, the media processing system 125 receives raw video 
data from the video recorder 330 and performs any necessary conversion, encoding, and 
compression on the video data. 
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[0028] In another embodiment, the printer 100 includes an embedded audio recorder 
340. In this embodiment, the external source of media data is a series of sounds that are 
converted into an electrical format by the embedded audio recorder 340. The audio 
recorder 340 converts the recorded sound signal into a suitable electrical format, such as 
that described in MPEG-2. Optionally, an analog-to-digital converter 335 converts the 
sound directly from the audio recorder 340 into a digital format, which is then provided to 
the media processing system 125. In yet another embodiment, the media processing 
system 125 receives raw audio data from the audio recorder 340 and performs any 
. necessary conversion, encoding, and compression on the audio data. 
[0029] In another embodiment, the printer 100 comprises a media broadcast receiver 
345. The media broadcast receiver 345 can be tuned to a media broadcast from a media 
broadcaster. Types of media broadcasts that can be tuned into and received include 
broadcasts from the Internet, cable, television, radio, and any other broadcasted media 
source. To tune into a particular media broadcast, a user may select the broadcast using 
the user interface 1 10. In one embodiment, the broadcast receiver 345 is an embedded 
TV receiver, an embedded radio receiver, an embedded short-wave radio receiver, an 
embedded satellite radio receiver, an embedded two-way radio, an embedded cellular 
phone, or a combination of any of the above. However, these are not meant to be 
limiting, as the broadcast receiver 345 can be any device that is capable of tuning into a 
signal and receiving media data therefrom. 

[0030] In another embodiment, the printer 100 includes an embedded event sensor 
350 that receives a signal about or otherwise observes a real- world event. Similar to a 
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broadcast receiver 345, the event sensor 350 may receive data about a real- world event, 
for example, by an embedded National Weather Service radio alert receiver or an 
embedded TV Emergency Broadcast System (EBS) alert monitor. Alternatively, the 
event sensor 350 may comprise an embedded heat sensor, an embedded humidity sensor, 
or another type of measuring device so that the event sensor 350 can directly measure the 
environment and generate time-based media therefrom. In this way, the printer 100 can 
be used to record events near the printer or far from it and generate meaningful time- 
based informational outputs based on those events. 

[0031] In another embodiment, the printer may include video capture hardware 355. 
In one embodiment, the video capture hardware 355 is designed to be coupled to a 
computing system by a video cable thereof. The video cable from a display is attached to 
the printer 100, where the video signal is split with one signal directed to the computing 
system and another signal to the video capture hardware 355. The video capture 
hardware 355 performs a differencing between successive frames of the video signal and 
saves frames with a difference that exceeds a threshold on a secondary storage in the 
printer 100. This offloads such processing from the computing system, thereby 
improving responsiveness and user experience and providing an easily browseable record 
of a user's activities during the day. To take advantage of the printing capabilities of the 
multifunction printer, the user can chose to print selected frames captured by the video 
capture hardware 355. The printing can be generated on demand with the user interface 
1 10 on the printer or from the attached computing system, or automatically with 
scheduling software. In this way, a user can view a replay of any actions taken on the 
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computing system. Notably, the captured content can be effectively compressed because 
the differences between frames are small. 

[0032] In another embodiment, the video capture hardware 355 is coupled to a 
converter module 360, such as VGA-to-NTSC conversion hardware. Such an 
embodiment could be used in conjunction with a projector to capture presentations made 
with the projector. Audio capture could also be employed to record a speaker's oral 
presentation. To use the video capture hardware 355 in this way, a user could connect a 
laptop or other computing system and the projector to the printer 100. The printer 100 
then captures video frames, compares them to the most recently captured frame, and 
retains those frames that are different. A parallel audio track may also be saved. This 
capability could also be used in a desktop printer to record a presentation made on a 
computing system connected to the printer. The printer can then serve the audio or it can 
be written to a digital medium, such as an SD disk that can be played from a cell phone or 
a PDA. The audio could also be written to a bar code on a printed representation. 
[0033] In another embodiment, the printer 100 comprises an ultrasonic pen capture 
device 365. In this embodiment, the printer 100 includes a hardware module 365 that 
serves as a low-cost sensor that can be coupled to a paper document. With such a device, 
a user can write on a paper document, and the results are saved on the printer 100. In one 
form, the captured results include the user's writing in combination with time data that 
indicates when the writing occurred. This time-based media data can then be printed or 
sent (e.g., by electronic mail) to someone else, for example showing a user's notes as well 
as time stamps that indicate when the notes were taken. 
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Printed Output System ! 
[0034] The printed output system 115 may comprise any standard printing hardware, 
including that found in standard laser printers, inkjet printers, thermal wax transfer 
printers, dye sublimation printers, dot matrix printers, plotters, or any other type of 
printing mechanisms suitable for creating a printer image on an appropriate physical 
medium. In the example described herein, a laser printer mechanism is described; 
however, it should be understood that any suitable printing system can be used. The 
printer 100 includes any necessary subsystems, as known by one skilled in the art, to print 
on a printable medium, such as a sheet of paper. 

[0035] In one embodiment, the printed output system 115 comprises a media supply 
handler 405 that receives blank paper to be printed on. The media supply handler 405 
typically obtains the paper from a supply tray 410. The printer 100 may include multiple 
supply trays 410, allowing the printer to accommodate different sizes and types of paper 
as well as trays 410 of varying capacity. When the printer 100 needs blank paper for 
printing, the media supply handler 405 provides the print engine 420 with a sheet of blank 
medium. 

[0036] The formatter 415 converts data received from the media processing system 
125 into a format that the print engine 420 can use to create an image on the paper. The 
print engine 420 creates an image on the paper as indicated by the formatter 415. A fuser 
425 then uses high temperature and pressure to fuse the image onto the paper to fix the 
image thereon. Once the image is fixed, the paper is fed to the media output handler 430. 

I 
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Although not shown, it is appreciated that the printer 100 includes any necessary motors, 
gears, and diverters to cause the paper to move through the printer 100. 
[0037] The media output handler 430 receives one or more printed sheets of paper 
and performs any requested finishing to the sheets. For example, the media output 
handler 430 may include a sorter 435 to sort or collate the sheets for multiple copies and a 
stapler 440 to attach the sheets together. When the finishing process is complete, the 
media output handler 430 moves the sheets to an output tray 445, of which there may be 
multiple trays 445 to accommodate different sizes, types, and capacities of printed output. 

Electronic Output System 
[0038] The electronic output system 120 can be designed to produce an electronic 
output related to the media data in any desired format. Because of the great variety of 
types and formats of electronic outputs, the electronic output system 120 may take any of 
a number of forms for producing an electronic output desired by the user. FIG. 5 
illustrates some examples of different embodiments of the electronic output system 105. 
In particular implementations, the printer 100 may have only one or only a subset of the 
various components shown, and in addition it may have other types of not shown. 
[0039] In one embodiment, the printer 100 writes the electronic output to a removable 
media device with a media writer 505. Many different types of media writers are know in 
the art, and the media writer 505 many comprise any of these. For example, the media 
writer 505 may be configured to write the electronic output to removable storage devices 
such as a writeable DVD or CD, a video cassette tape, an audio cassette tape, a flash card, 

i 

i 

a computer disk, an SD disk, a memory stick, or any other appropriate electronically , 

i 
i 
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readable medium. Moreover, the electronic output system 120 may include a number of 
media writers 505 of different types to allow the printer 100 to print onto different 
electronic formats. In addition, the electronic output system 120 may include a number of 
media writers 505 of the same type to increase the output capacity of the printer 100. 
[0040] The removable storage device that receives the electronic output from the 
printer 100 may be fed to the media writer directly by a user, for example by inserting a 
blank disk into a drive. In another embodiment, the printer 100 includes an electronic 
media handling mechanism 510 coupled to the media writer 505 that automatically 
provides the media writer 505 with an appropriate type of removable storage device. The 
handling mechanism 510 may further be configured to physically place written-to storage 
devices into an output tray 515. In one embodiment, a series of blank storage devices are 
fed to the printer 100 by a bandolier 520 or other type of feeder, allowing the printer 100 
to create a high volume of electronic output without requiring a significant amount of 
interaction with a human operator. The bandolier 520 preferably then places the written 
to devices into an output tray 515. 

[0041] In another embodiment, the media writer 505 is a disposable media writer, 
configured to write electronic data to a disposable removable media storage mechanism. 
In another embodiment, the media writer 505 writes the electronic data to a self- 
destructing medium. In this way, a user can view the electronic data for a predetermined 
number of times or during a predetermined period of time, after which the electronic data 
are no longer viewable. 
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[0042] In another embodiment, the electronic output system 120 includes a speaker 
system 530. The speaker system 530 is designed to receive an audio signal from the 
media processing system 125, in response to which the audio is played from an embedded 
speaker 530 in the printer 100. The electronic output system 120 may further include a 
player 525 or audio renderer that receives an encoded audio signal from the media 
processing system 125 and converts it into an audio signal for the speaker 530. The 
player 525 thus takes some of the processing load off the media processing system 125. 
For example, the player 525 may include a MIDI player for generating the audio signal; 
however, many other audio Tenderers may be used, in either hardware or software. 
[0043] In another embodiment, the electronic output system 120 includes a video 
display 535. The video display 535is designed to receive a video signal from the media 
processing system 125, in response to which the video is played on the video display 535 
embedded into the printer 100. Similarly, the video display 535 may receive the video 
signal directly from a driver to reduce the processing load on the media processing system 
125. 

[0044] In another embodiment, the printer 100 transmits the electronic output that is 
to be printed to another device as a signal. This signal can later be fixed in a tangible 
medium by the external device. To facilitate this, the electronic output system 120 
includes a communication interface 540. The communication interface receives the 
electronic output from the media processing system 125 and sends the electronic output to 
the external device, which may be in communication with the printer 100 over a local 
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network, the Internet, a wireless network, Universal Serial Bus (USB), IEEE 1394, a 
direct connection, or any other suitable communication means. 
[0045] In another embodiment, the electronic output system 120 comprises an 
embedded web page display 545. The web page display 545 allows a user to see a 
representation of the electronic output in a web-based form. 

Media Processing System 
[0046] The media processing system 125 of the printer 100 is designed to perform the 
specialized functionality of the multifunction printer 100. To send and receive messages 
between external devices or the user interface 1 10, the processing system 125 includes a 
processor 135 and a memory 130. In addition, the media processing system includes one 
or more hardware and/or software modules that enable the printer 100 to create related 
printed and electronic outputs for different types of time-based media. In this way, the 
printer 100 can be configured to have any number of embedded functionalities. 
[0047] In one embodiment, the printer includes one or a combination of an embedded 
multimedia server module 610, an encryption module 620, an audio localization module 
630, a video motion detection module 640, a media format conversion module 650, a 
content-based processing module 660, and an A/V content recognition module 670. Each 
of these embedded functionality modules are described in further detail below. 
[0048] Although the media processing system 125 is configured to perform at least 
some of the processing of the media data on the printer 100, the system 125 may be 
coupled to an external computing device 680, which can share some of the computing 
burden. The shared computing device 680 may be a computer system or a dedicated 
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media processing hardware device. In this way, the printer 100 relieves the source of the 
time-based media from at least some of the processing burden required to implement the 
printer's functionality, but the printer 100 need not shoulder the entire burden. The 
printer 100 can thus avoid slow-downs that can result from a heavy processing load, 
which may be especially important for shared printers. 

Printer With Multimedia Functionality 
[0049] As explained, the printer 100 may include an embedded multimedia server 
module 610 that enables the printer 100 to act as a multimedia server and have associated 
functionality. In various embodiments, the multimedia server module 610 includes 
hardware and software for carrying out multimedia functionality, media processing 
software, and computer interface hardware and software. In this way, the printer 100 can 
act like a multimedia server, which could be shared by a work group or used as a personal 
printer. Various embodiments of a multifunction printer having multimedia functionality 
are possible applications for a printer in accordance with embodiments of this invention. 
A number of specific embodiments for such a printer are described in a co-pending U.S. 
patent application entitled, "Printer with Multimedia Server," filed March 30, 2004, 
Attorney Docket No. 20412-08351, which application is incorporated by reference in its 
entirety. 

Printer With Encryption Functionality 
[0050] Similarly, the printer 100 may include an embedded encryption module 620, 
which enables the printer 100 to perform encryption functions on the received media data. 
In a typical application, the printer 100 receives a media file to be encrypted, encrypts the 
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file, and then writes the encrypted media file onto a removable storage device. The 
printer 100 also prints an associated paper output that provides information about the 
encryption, such as the encryption key, a password, an identification of the media, and 
optionally a description of the content of the media encrypted. 
[0051] While various embodiments of a multifunction printer having encryption 
functionality are possible applications for a printer in accordance with embodiments of 
this invention. A number of specific embodiments for such a printer are described in a 
co-pending U.S. patent application entitled, "Multimedia Output Device Having 
Embedded Encryption Functionality," filed March 30, 2004, Attorney Docket No. 20412- 
08352, which application is incorporated by reference in its entirety. 

Printer With Audio Localization and/or Video Motion Detection Functionality 
[0052] In addition, the printer 100 may include an embedded audio localization 
module 630 and/or an embedded video motion detection module 640, which enable the 
printer 100 to produce outputs based on the location of sound or the detection of motion 
in the received media. In this way, the printer can observe events and aid the user in 
filtering through a massive amount of data about those events. 
[0053] In one embodiment, the printer 100 includes embedded audio capture and 
sound source localization hardware. In another embodiment, or in addition, the printer 
100 includes video capture and motion detection hardware and software. In this way, the 
printer 100 can create a printed document and an electronic output that, together, 
summarize the audio and/or video events that were detected. The events can be defined 
and their representation in a document can be designed before the events occur. That 
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information can then be communicated to the printer 100 by printing the document before 
the events happen. The printer can recognize future events in the documents it receives 
and schedule documents for printing after the events are detected. This may be helpful, 
for example, to verify the creation of intellectual property or to detect the time when a 
burglar entered an office. If the documents are printed, the paper can be easily 
incorporated in an existing paper-based workflow. 

[0054] In an example embodiment, the media data include video data with a separate 
multi-channel audio track that was produced from a known configuration of microphones. 
In this way, the system can be designed to optimize audio source localization. The printer 
can then create outputs based on the result of sound localization and user input that 
indicates the directions of interest to the user. For example, clips can be selected from the 
video when people in those locations were speaking. Key frames and bar codes for those 
clips can then be printed on the printed output. 

[0055] Various embodiments of a multifunction printer having audio localization 
and/or video motion detection functionality are possible applications for a printer in 
accordance with embodiments of this invention. A number of specific embodiments for 
such a printer are described in a co-pending U.S. patent application entitled, "Printer With 
Audio/Video Localization," filed March 30, 2004, Attorney Docket No. 20412-08356, 
which application is incorporated by reference in its entirety. 

Printer With Media Format Conversion Functionality 
[0056] As explained above, the printer 100 may include an embedded media format 
conversion module 650. The media format conversion module 650 enables the printer 
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100 to convert the received media data into various formats other than that in which it is 
received. The media format conversion module 650 may include appropriate hardware 
and software for carrying out its functionality, such as audio rendering software, 
processing logic for different types of media conversions, and a speaker output. In this 
way, the printer 100 may contain an embedded formatter that can decode, encode, and/or 
convert media data from one format type into another format type. 
[0057] In one embodiment, the printer 100 converts a digital audio file, such as a 
MIDI file, into a paper representation as a musical score. Optionally, the printer 100 also 
converts the input data to another audio format, such as MP3, that could be played on 
other devices, such as a cell phone. Bar codes can be added to a printed paper 
representation of the media to let the user play the audio corresponding to lines of music 
on a remote device or through the speaker on the printer. 

[0058] Various other embodiments of a multifunction printer having audio format 
conversion functionality are possible applications for a printer in accordance with 
embodiments of this invention. A number of specific embodiments for such a printer are 
described in a co-pending U.S. patent application entitled, "Music Processing Printer," 
filed March 30, 2004, Attorney Docket No. 20412-08358, which application is 
incorporated by reference in its entirety. 

Printer With Content-Based Processing Functionality 
[0059] As explained, the printer 100 may include an embedded content-based 
processing module 660, which enables the printer 100 to receive and process real-time 
broadcasts of media data responsive to the content of the media. In one embodiment, the 
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printer 100 includes an embedded receiver, such as a television, satellite, or cable, radio, 
or other type of media receiver. The printer 100 may further include processing logic that 
can monitor a live media feed, recording media, generate a printed representation of the 
media, and play the audio or video on the printer's console. The printer 100 thus enables 
the live monitoring of the content of radio or television broadcasts and the generation of 
paper and storage of multimedia in response to events detected in those streams. 
[0060] Examples of events include tone sequences indicative of National Weather 
Service or Emergency Alert Service alerts, but they could also include user-defined 
events such as the appearance of a specified set of key words in the closed caption of a 
television broadcast, the appearance of a given image in a video stream (e.g., a face 
image), or the occurrence of gun shots in an audio stream. The instant generation of the 
paper when events occur allows the user to pick up a print-out off the printer at any later 
time without needing to press any other buttons or performing other manual operations. 
[0061] Various embodiments of a multifunction printer having content-based 
processing functionality are possible applications for a printer in accordance with 
embodiments of this invention. A number of specific embodiments for such a printer are 
described in a co-pending U.S. patent application entitled, "Printer With Audio or Video 
Receiver, Recorder, and Real-Time Content-Based Processing Logic," filed March 30, 
2004, Attorney Docket No. 20412-08369, which application is incorporated by reference 
in its entirety. 
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Printer With Audio /Video Content Recognition and Processing Functionality 
[0062] As explained, the printer 100 may include an embedded A/V content 
recognition module 670, which enables the printer 100 to recognize particular properties 
in the content of the media data and process the data based on that recognized content. In 
one embodiment, the printer 100 receives the media data from an electronic file, which 
the printer 100 uses to convert the time-based media into related printed and electronic 
outputs. In one embodiment, the printer 100 includes audio and/or video content 
recognition and processing software and/or hardware. The types of embedded hardware 
and software required depend on the desired content recognition functionality, and 
examples of the types of functionalities the embedded A/V content recognition module 
370 may perform include video event detection, video foreground/background 
segmentation, face detection, face image matching, face recognition, face cataloging, 
video text localization, video optical character recognition (OCR), language translation, 
frame classification, clip classification, image stitching, audio reformatting, speech 
recognition, audio event detection, audio waveform matching, caption alignment, audio- 
caption alignment, and any other type of media content recognition algorithms. The 
printer 100 further includes processing logic that interacts with the user through the user 
interface 110 to generate the desired printed and electronic outputs. 
[0063] Various embodiments of a multifunction printer having audio/video content 
recognition and processing functionality are possible applications for a printer in 
accordance with embodiments of this invention. A number of specific embodiments for 
such a printer are described in a co-pending U.S. patent application entitled, "Printing 
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System With Embedded Audio/Video Content Recognition and Processing," filed March 
30, 2004, Attorney Docket No. 20412-08394, which application is incorporated by 
reference in its entirety. 
General Comments 

[0064] While examples of suitable printing systems are described above, the 
description of the printer and its document production means is not meant to be limiting. 
Depending on the intended application, a printer can take many different forms other than 
the typical office or home-use printer with which most people are familiar. Therefore, it 
should be understood that the definition of a printer includes any device that is capable of 
producing an image, words, or any other markings on a surface. Although printing on 
paper is discussed above, it should be understood that a printer in accordance with 
various embodiments of the present invention could produce an image, words, or other 
markings onto a variety of tangible media, such as transparency sheets for overhead 
projectors, film, slides, canvass, glass, stickers, or any other medium that accepts such 
markings. 

[0065] In addition, the description and use of media and media data are not meant to 
be limiting, as media include any information, tangible or intangible, used to represent 
any kind of media or multimedia content, such as all or part of an audio and/or video file, 
a data stream having media content, or a transmission of media content. Media may 
include one or a combination of audio (including music, radio broadcasts, recordings, 
advertisements, etc.), video (including movies, video clips, television broadcasts, 
advertisements, etc.), software (including video games, multimedia programs, graphics 
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software, etc.), and pictures; however, this listing is not exhaustive. Furthermore, media 
and media data may further include anything that itself comprises media or media data, in 
whole or in part, and media data includes data that describes a real-world event. Media 
data can be encoded using any encoding technology, such as MPEG in the case of video 
and MP3 in the case of audio. They may also be encrypted to protect their content using 
an encryption algorithm, such as DES, triple DES, or any other suitable encryption 
technique. 

[0066] Moreover, any of the steps, operations, or processes described herein can be 
performed or implemented with one or more software modules or hardware modules, 
alone or in combination with other devices. It should further be understood that portions 
of the printer described in terms of hardware elements may be implemented with 
software, and that software elements may be implemented with hardware, such as hard- 
coded into a dedicated circuit. In one embodiment, a software module is implemented 
with a computer program product comprising a computer-readable medium containing 
computer program code, which can be executed by a computer processor for performing 
the steps, operations, or processes described herein. 

[0067] In alternative embodiments, the printer can use multiple application servers, 
acting in cooperation. Any of the requests or messages sent or received by the printer can 
be sent across a network, using local cables such as IEEE1394, Universal Serial Bus, 
using wireless networks such as IEEE 802.1 1 or IEEE 802.15 networks, or in any 
combination of the above. 
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[0068] The foregoing description of the embodiments of the invention has been 
presented for the purpose of illustration; it is not intended to be exhaustive or to limit the 
invention to the precise forms disclosed. Persons skilled in the relevant art can appreciate 
that many modifications and variations are possible in light of the above teachings. It is 
therefore intended that the scope of the invention be limited not by this detailed 
description, but rather by the claims appended hereto. 



-29- 



20412/08340/SF/5116435. 1 



